{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the third blog post of [Object Detection with YOLO blog series](https://fairyonice.github.io/tag/object-detection-using-yolov2-on-pascal-voc2012-series.html). This blog discusses the YOLO's model architecture. I will use PASCAL VOC2012 data. This blog assumes that the readers have read the previous two blog posts - [Part 1](https://fairyonice.github.io/Part_1_Object_Detection_with_Yolo_for_VOC_2014_data_anchor_box_clustering.html), [Part 2](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html).\n",
    "\n",
    "\n",
    "## Andrew Ng's YOLO lecture\n",
    "- [Neural Networks - Bounding Box Predictions](https://www.youtube.com/watch?v=gKreZOUi-O0&t=0s&index=7&list=PL_IHmaMAvkVxdDOBRg2CbcJBq9SY7ZUvs)\n",
    "- [C4W3L06 Intersection Over Union](https://www.youtube.com/watch?v=ANIzQ5G-XPE&t=7s)\n",
    "- [C4W3L07 Nonmax Suppression](https://www.youtube.com/watch?v=VAo84c1hQX8&t=192s)\n",
    "- [C4W3L08 Anchor Boxes](https://www.youtube.com/watch?v=RTlwl2bv0Tg&t=28s)\n",
    "- [C4W3L09 YOLO Algorithm](https://www.youtube.com/watch?v=9s_FpMpdYW8&t=34s)\n",
    "\n",
    "## Reference\n",
    "- [You Only Look Once:Unified, Real-Time Object Detection](https://arxiv.org/pdf/1506.02640.pdf) \n",
    "\n",
    "- [YOLO9000:Better, Faster, Stronger](https://arxiv.org/pdf/1612.08242.pdf)\n",
    " \n",
    "- [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2)\n",
    "\n",
    "## Reference in my blog\n",
    "- [Part 1 Object Detection using YOLOv2 on Pascal VOC2012 - anchor box clustering](https://fairyonice.github.io/Part_1_Object_Detection_with_Yolo_for_VOC_2014_data_anchor_box_clustering.html)\n",
    "- [Part 2 Object Detection using YOLOv2 on Pascal VOC2012 - input and output encoding](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html)\n",
    "- [Part 3 Object Detection using YOLOv2 on Pascal VOC2012 - model](https://fairyonice.github.io/Part_3_Object_Detection_with_Yolo_using_VOC_2012_data_model.html)\n",
    "- [Part 4 Object Detection using YOLOv2 on Pascal VOC2012 - loss](https://fairyonice.github.io/Part_4_Object_Detection_with_Yolo_using_VOC_2012_data_loss.html)\n",
    "- [Part 5 Object Detection using YOLOv2 on Pascal VOC2012 - training](https://fairyonice.github.io/Part_5_Object_Detection_with_Yolo_using_VOC_2012_data_training.html)\n",
    "- [Part 6 Object Detection using YOLOv2 on Pascal VOC 2012 data - inference on image](https://fairyonice.github.io/Part_6_Object_Detection_with_Yolo_using_VOC_2012_data_inference_image.html)\n",
    "- [Part 7 Object Detection using YOLOv2 on Pascal VOC 2012 data - inference on video](https://fairyonice.github.io/Part_7_Object_Detection_with_Yolo_using_VOC_2012_data_inference_video.html)\n",
    "\n",
    "## My GitHub repository \n",
    "This repository contains all the ipython notebooks in this blog series and the funcitons (See backend.py). \n",
    "- [FairyOnIce/ObjectDetectionYolo](https://github.com/FairyOnIce/ObjectDetectionYolo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.6.3 |Anaconda, Inc.| (default, Oct  6 2017, 12:04:38) \n",
      "[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import os\n",
    "import sys\n",
    "print(sys.version)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define anchor box\n",
    "<code>ANCHORS</code> defines the number of anchor boxes and the shape of each anchor box.\n",
    "The choice of the anchor box specialization is already discussed in [Part 1 Object Detection using YOLOv2 on Pascal VOC2012 - anchor box clustering](https://fairyonice.github.io/Part_1_Object_Detection_with_Yolo_for_VOC_2014_data_anchor_box_clustering.html). \n",
    "\n",
    "Based on the K-means analysis in the previous blog post, I will select 4 anchor boxes of following width and height. The width and heights are rescaled in the grid cell scale (Assuming that the number of grid size is 13 by 13.) See [Part 2 Object Detection using YOLOv2 on Pascal VOC2012 - input and output encoding](https://fairyonice.github.io/Part%202_Object_Detection_with_Yolo_using_VOC_2014_data_input_and_output_encoding.html) to learn how I rescal the anchor box shapes into the grid cell scale.\n",
    "\n",
    "Here I choose 4 anchor boxes. With 13 by 13 grids, every frame gets 4 x 13 x 13 = 676 bouding box predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ANCHORS = np.array([1.07709888,  1.78171903,  # anchor box 1, width , height\n",
    "                    2.71054693,  5.12469308,  # anchor box 2, width,  height\n",
    "                   10.47181473, 10.09646365,  # anchor box 3, width,  height\n",
    "                    5.48531347,  8.11011331]) # anchor box 4, width,  height"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Label vector containing 20 object classe names."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "LABELS = ['aeroplane',  'bicycle', 'bird',  'boat',      'bottle', \n",
    "          'bus',        'car',      'cat',  'chair',     'cow',\n",
    "          'diningtable','dog',    'horse',  'motorbike', 'person',\n",
    "          'pottedplant','sheep',  'sofa',   'train',   'tvmonitor']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## YOLOv2 Model Architecture\n",
    "While YOLO's input and output encodings are complex, and loss function of YOLO is quite complex (which will be discussed very soon), the model architecture is simple. \n",
    "It repeatedly stacks Convolusion + Batch Normalization + Leaky Relu layers until the image shape reduces to the grid cell size. \n",
    "Here is the model defenition, extracted from [experiencor/keras-yolo2](https://github.com/experiencor/keras-yolo2). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yumikondo/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_image (InputLayer)        (None, 416, 416, 3)  0                                            \n",
      "__________________________________________________________________________________________________\n",
      "conv_1 (Conv2D)                 (None, 416, 416, 32) 864         input_image[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "norm_1 (BatchNormalization)     (None, 416, 416, 32) 128         conv_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_1 (LeakyReLU)       (None, 416, 416, 32) 0           norm_1[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "maxpool1_416to208 (MaxPooling2D (None, 208, 208, 32) 0           leaky_re_lu_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv_2 (Conv2D)                 (None, 208, 208, 64) 18432       maxpool1_416to208[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "norm_2 (BatchNormalization)     (None, 208, 208, 64) 256         conv_2[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_2 (LeakyReLU)       (None, 208, 208, 64) 0           norm_2[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "maxpool1_208to104 (MaxPooling2D (None, 104, 104, 64) 0           leaky_re_lu_2[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv_3 (Conv2D)                 (None, 104, 104, 128 73728       maxpool1_208to104[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "norm_3 (BatchNormalization)     (None, 104, 104, 128 512         conv_3[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_3 (LeakyReLU)       (None, 104, 104, 128 0           norm_3[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv_4 (Conv2D)                 (None, 104, 104, 64) 8192        leaky_re_lu_3[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "norm_4 (BatchNormalization)     (None, 104, 104, 64) 256         conv_4[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_4 (LeakyReLU)       (None, 104, 104, 64) 0           norm_4[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv_5 (Conv2D)                 (None, 104, 104, 128 73728       leaky_re_lu_4[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "norm_5 (BatchNormalization)     (None, 104, 104, 128 512         conv_5[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_5 (LeakyReLU)       (None, 104, 104, 128 0           norm_5[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "maxpool1_104to52 (MaxPooling2D) (None, 52, 52, 128)  0           leaky_re_lu_5[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv_6 (Conv2D)                 (None, 52, 52, 256)  294912      maxpool1_104to52[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "norm_6 (BatchNormalization)     (None, 52, 52, 256)  1024        conv_6[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_6 (LeakyReLU)       (None, 52, 52, 256)  0           norm_6[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv_7 (Conv2D)                 (None, 52, 52, 128)  32768       leaky_re_lu_6[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "norm_7 (BatchNormalization)     (None, 52, 52, 128)  512         conv_7[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_7 (LeakyReLU)       (None, 52, 52, 128)  0           norm_7[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv_8 (Conv2D)                 (None, 52, 52, 256)  294912      leaky_re_lu_7[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "norm_8 (BatchNormalization)     (None, 52, 52, 256)  1024        conv_8[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_8 (LeakyReLU)       (None, 52, 52, 256)  0           norm_8[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "maxpool1_52to26 (MaxPooling2D)  (None, 26, 26, 256)  0           leaky_re_lu_8[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "conv_9 (Conv2D)                 (None, 26, 26, 512)  1179648     maxpool1_52to26[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "norm_9 (BatchNormalization)     (None, 26, 26, 512)  2048        conv_9[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_9 (LeakyReLU)       (None, 26, 26, 512)  0           norm_9[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "conv_10 (Conv2D)                (None, 26, 26, 256)  131072      leaky_re_lu_9[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "norm_10 (BatchNormalization)    (None, 26, 26, 256)  1024        conv_10[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_10 (LeakyReLU)      (None, 26, 26, 256)  0           norm_10[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_11 (Conv2D)                (None, 26, 26, 512)  1179648     leaky_re_lu_10[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_11 (BatchNormalization)    (None, 26, 26, 512)  2048        conv_11[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_11 (LeakyReLU)      (None, 26, 26, 512)  0           norm_11[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_12 (Conv2D)                (None, 26, 26, 256)  131072      leaky_re_lu_11[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_12 (BatchNormalization)    (None, 26, 26, 256)  1024        conv_12[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_12 (LeakyReLU)      (None, 26, 26, 256)  0           norm_12[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_13 (Conv2D)                (None, 26, 26, 512)  1179648     leaky_re_lu_12[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_13 (BatchNormalization)    (None, 26, 26, 512)  2048        conv_13[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_13 (LeakyReLU)      (None, 26, 26, 512)  0           norm_13[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "maxpool1_26to13 (MaxPooling2D)  (None, 13, 13, 512)  0           leaky_re_lu_13[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv_14 (Conv2D)                (None, 13, 13, 1024) 4718592     maxpool1_26to13[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "norm_14 (BatchNormalization)    (None, 13, 13, 1024) 4096        conv_14[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_14 (LeakyReLU)      (None, 13, 13, 1024) 0           norm_14[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_15 (Conv2D)                (None, 13, 13, 512)  524288      leaky_re_lu_14[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_15 (BatchNormalization)    (None, 13, 13, 512)  2048        conv_15[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_15 (LeakyReLU)      (None, 13, 13, 512)  0           norm_15[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_16 (Conv2D)                (None, 13, 13, 1024) 4718592     leaky_re_lu_15[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_16 (BatchNormalization)    (None, 13, 13, 1024) 4096        conv_16[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_16 (LeakyReLU)      (None, 13, 13, 1024) 0           norm_16[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_17 (Conv2D)                (None, 13, 13, 512)  524288      leaky_re_lu_16[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_17 (BatchNormalization)    (None, 13, 13, 512)  2048        conv_17[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_17 (LeakyReLU)      (None, 13, 13, 512)  0           norm_17[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_18 (Conv2D)                (None, 13, 13, 1024) 4718592     leaky_re_lu_17[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_18 (BatchNormalization)    (None, 13, 13, 1024) 4096        conv_18[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_18 (LeakyReLU)      (None, 13, 13, 1024) 0           norm_18[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_19 (Conv2D)                (None, 13, 13, 1024) 9437184     leaky_re_lu_18[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "norm_19 (BatchNormalization)    (None, 13, 13, 1024) 4096        conv_19[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_21 (Conv2D)                (None, 26, 26, 64)   32768       leaky_re_lu_13[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_19 (LeakyReLU)      (None, 13, 13, 1024) 0           norm_19[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "norm_21 (BatchNormalization)    (None, 26, 26, 64)   256         conv_21[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_20 (Conv2D)                (None, 13, 13, 1024) 9437184     leaky_re_lu_19[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_21 (LeakyReLU)      (None, 26, 26, 64)   0           norm_21[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "norm_20 (BatchNormalization)    (None, 13, 13, 1024) 4096        conv_20[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lambda_1 (Lambda)               (None, 13, 13, 256)  0           leaky_re_lu_21[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_20 (LeakyReLU)      (None, 13, 13, 1024) 0           norm_20[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 13, 13, 1280) 0           lambda_1[0][0]                   \n",
      "                                                                 leaky_re_lu_20[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "conv_22 (Conv2D)                (None, 13, 13, 1024) 11796480    concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "norm_22 (BatchNormalization)    (None, 13, 13, 1024) 4096        conv_22[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_22 (LeakyReLU)      (None, 13, 13, 1024) 0           norm_22[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "conv_23 (Conv2D)                (None, 13, 13, 100)  102500      leaky_re_lu_22[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "final_output (Reshape)          (None, 13, 13, 4, 25 0           conv_23[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "input_hack (InputLayer)         (None, 1, 1, 1, 50,  0                                            \n",
      "__________________________________________________________________________________________________\n",
      "hack_layer (Lambda)             (None, 13, 13, 4, 25 0           final_output[0][0]               \n",
      "                                                                 input_hack[0][0]                 \n",
      "==================================================================================================\n",
      "Total params: 50,650,436\n",
      "Trainable params: 102,500\n",
      "Non-trainable params: 50,547,936\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from keras.models import Sequential, Model\n",
    "from keras.layers import Reshape, Activation, Conv2D, Input, MaxPooling2D, BatchNormalization, Flatten, Dense, Lambda\n",
    "from keras.layers.advanced_activations import LeakyReLU\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint, TensorBoard\n",
    "from keras.optimizers import SGD, Adam, RMSprop\n",
    "from keras.layers.merge import concatenate\n",
    "import keras.backend as K\n",
    "import tensorflow as tf\n",
    "\n",
    "# the function to implement the orgnization layer (thanks to github.com/allanzelener/YAD2K)\n",
    "def space_to_depth_x2(x):\n",
    "    return tf.space_to_depth(x, block_size=2)\n",
    "\n",
    "def ConvBatchLReLu(x,filters,kernel_size,index,trainable):\n",
    "    # when strides = None, strides = pool_size.\n",
    "    x = Conv2D(filters, kernel_size, strides=(1,1), \n",
    "               padding='same', name='conv_{}'.format(index), \n",
    "               use_bias=False, trainable=trainable)(x)\n",
    "    x = BatchNormalization(name='norm_{}'.format(index), trainable=trainable)(x)\n",
    "    x = LeakyReLU(alpha=0.1)(x)\n",
    "    return(x)\n",
    "def ConvBatchLReLu_loop(x,index,convstack,trainable):\n",
    "    for para in convstack:\n",
    "        x = ConvBatchLReLu(x,para[\"filters\"],para[\"kernel_size\"],index,trainable)\n",
    "        index += 1\n",
    "    return(x)\n",
    "def define_YOLOv2(IMAGE_H,IMAGE_W,GRID_H,GRID_W,TRUE_BOX_BUFFER,BOX,CLASS, trainable=False):\n",
    "    convstack3to5  = [{\"filters\":128, \"kernel_size\":(3,3)},  # 3\n",
    "                      {\"filters\":64,  \"kernel_size\":(1,1)},  # 4\n",
    "                      {\"filters\":128, \"kernel_size\":(3,3)}]  # 5\n",
    "                    \n",
    "    convstack6to8  = [{\"filters\":256, \"kernel_size\":(3,3)},  # 6\n",
    "                      {\"filters\":128, \"kernel_size\":(1,1)},  # 7\n",
    "                      {\"filters\":256, \"kernel_size\":(3,3)}]  # 8\n",
    "    \n",
    "    convstack9to13 = [{\"filters\":512, \"kernel_size\":(3,3)},  # 9\n",
    "                      {\"filters\":256, \"kernel_size\":(1,1)},  # 10\n",
    "                      {\"filters\":512, \"kernel_size\":(3,3)},  # 11\n",
    "                      {\"filters\":256, \"kernel_size\":(1,1)},  # 12\n",
    "                      {\"filters\":512, \"kernel_size\":(3,3)}]  # 13\n",
    "        \n",
    "    convstack14to20 = [{\"filters\":1024, \"kernel_size\":(3,3)}, # 14 \n",
    "                       {\"filters\":512,  \"kernel_size\":(1,1)}, # 15\n",
    "                       {\"filters\":1024, \"kernel_size\":(3,3)}, # 16\n",
    "                       {\"filters\":512,  \"kernel_size\":(1,1)}, # 17\n",
    "                       {\"filters\":1024, \"kernel_size\":(3,3)}, # 18\n",
    "                       {\"filters\":1024, \"kernel_size\":(3,3)}, # 19\n",
    "                       {\"filters\":1024, \"kernel_size\":(3,3)}] # 20\n",
    "    \n",
    "    input_image = Input(shape=(IMAGE_H, IMAGE_W, 3),name=\"input_image\")\n",
    "    true_boxes  = Input(shape=(1, 1, 1, TRUE_BOX_BUFFER , 4),name=\"input_hack\")    \n",
    "    # Layer 1\n",
    "    x = ConvBatchLReLu(input_image,filters=32,kernel_size=(3,3),index=1,trainable=trainable)\n",
    "    \n",
    "    x = MaxPooling2D(pool_size=(2, 2),name=\"maxpool1_416to208\")(x)\n",
    "    # Layer 2\n",
    "    x = ConvBatchLReLu(x,filters=64,kernel_size=(3,3),index=2,trainable=trainable)\n",
    "    x = MaxPooling2D(pool_size=(2, 2),name=\"maxpool1_208to104\")(x)\n",
    "    \n",
    "    # Layer 3 - 5\n",
    "    x = ConvBatchLReLu_loop(x,3,convstack3to5,trainable)\n",
    "    x = MaxPooling2D(pool_size=(2, 2),name=\"maxpool1_104to52\")(x)\n",
    "    \n",
    "    # Layer 6 - 8 \n",
    "    x = ConvBatchLReLu_loop(x,6,convstack6to8,trainable)\n",
    "    x = MaxPooling2D(pool_size=(2, 2),name=\"maxpool1_52to26\")(x) \n",
    "\n",
    "    # Layer 9 - 13\n",
    "    x = ConvBatchLReLu_loop(x,9,convstack9to13,trainable)\n",
    "        \n",
    "    skip_connection = x\n",
    "    x = MaxPooling2D(pool_size=(2, 2),name=\"maxpool1_26to13\")(x)\n",
    "    \n",
    "    # Layer 14 - 20\n",
    "    x = ConvBatchLReLu_loop(x,14,convstack14to20,trainable)\n",
    "\n",
    "    # Layer 21\n",
    "    skip_connection = ConvBatchLReLu(skip_connection,filters=64,\n",
    "                                     kernel_size=(1,1),index=21,trainable=trainable)\n",
    "    skip_connection = Lambda(space_to_depth_x2)(skip_connection)\n",
    "\n",
    "    x = concatenate([skip_connection, x])\n",
    "\n",
    "    # Layer 22\n",
    "    x = ConvBatchLReLu(x,filters=1024,kernel_size=(3,3),index=22,trainable=trainable)\n",
    "\n",
    "    # Layer 23\n",
    "    x = Conv2D(BOX * (4 + 1 + CLASS), (1,1), strides=(1,1), padding='same', name='conv_23')(x)\n",
    "    output = Reshape((GRID_H, GRID_W, BOX, 4 + 1 + CLASS),name=\"final_output\")(x)\n",
    "\n",
    "    # small hack to allow true_boxes to be registered when Keras build the model \n",
    "    # for more information: https://github.com/fchollet/keras/issues/2790\n",
    "    output = Lambda(lambda args: args[0],name=\"hack_layer\")([output, true_boxes])\n",
    "\n",
    "    model = Model([input_image, true_boxes], output)\n",
    "    return(model, true_boxes)\n",
    "\n",
    "IMAGE_H, IMAGE_W  = 416, 416\n",
    "GRID_H,  GRID_W   = 13 , 13\n",
    "TRUE_BOX_BUFFER   = 50\n",
    "BOX               = int(len(ANCHORS)/2)\n",
    "CLASS             = len(LABELS)\n",
    "## true_boxes is the tensor that takes \"b_batch\"\n",
    "model, true_boxes = define_YOLOv2(IMAGE_H,IMAGE_W,GRID_H,GRID_W,TRUE_BOX_BUFFER,BOX,CLASS, \n",
    "                                  trainable=False)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load pre-trained YOLOv2 weights \n",
    "Following the instruction at [YOLO: Real-Time Object Detection](https://pjreddie.com/darknet/yolov2/), we download the pre-trained weights using wget: \n",
    "\n",
    "<code>\n",
    "wget https://pjreddie.com/media/files/yolov2.weights\n",
    "</code>\n",
    "\n",
    "The weights are saved at:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "path_to_weight = \"./yolov2.weights\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following codes are extracted from [keras-yolo2/Yolo Step-by-Step.ipynb](https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "all_weights.shape = (50983565,)\n"
     ]
    }
   ],
   "source": [
    "class WeightReader:\n",
    "    # code from https://github.com/experiencor/keras-yolo2/blob/master/Yolo%20Step-by-Step.ipynb\n",
    "    def __init__(self, weight_file):\n",
    "        self.offset = 4\n",
    "        self.all_weights = np.fromfile(weight_file, dtype='float32')\n",
    "        \n",
    "    def read_bytes(self, size):\n",
    "        self.offset = self.offset + size\n",
    "        return self.all_weights[self.offset-size:self.offset]\n",
    "    \n",
    "    def reset(self):\n",
    "        self.offset = 4\n",
    "                \n",
    "weight_reader = WeightReader(path_to_weight)\n",
    "print(\"all_weights.shape = {}\".format(weight_reader.all_weights.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assign pre-trained weights to the following layers: \n",
    "<code> conv_i</code>, <code>norm_i</code>, <code> i = 1, 2,..., 22</code>. \n",
    "These layers do not depend on the number of object classes or the number of anchor boxes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def set_pretrained_weight(model,nb_conv, path_to_weight):\n",
    "    weight_reader = WeightReader(path_to_weight)\n",
    "    weight_reader.reset()\n",
    "    for i in range(1, nb_conv+1):\n",
    "        conv_layer = model.get_layer('conv_' + str(i)) ## convolusional layer\n",
    "\n",
    "        if i < nb_conv:\n",
    "            norm_layer = model.get_layer('norm_' + str(i)) ## batch normalization layer\n",
    "\n",
    "            size = np.prod(norm_layer.get_weights()[0].shape)\n",
    "\n",
    "            beta  = weight_reader.read_bytes(size)\n",
    "            gamma = weight_reader.read_bytes(size)\n",
    "            mean  = weight_reader.read_bytes(size)\n",
    "            var   = weight_reader.read_bytes(size)\n",
    "\n",
    "            weights = norm_layer.set_weights([gamma, beta, mean, var])       \n",
    "\n",
    "        if len(conv_layer.get_weights()) > 1: ## with bias\n",
    "            bias   = weight_reader.read_bytes(np.prod(conv_layer.get_weights()[1].shape))\n",
    "            kernel = weight_reader.read_bytes(np.prod(conv_layer.get_weights()[0].shape))\n",
    "            kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))\n",
    "            kernel = kernel.transpose([2,3,1,0])\n",
    "            conv_layer.set_weights([kernel, bias])\n",
    "        else: ## without bias\n",
    "            kernel = weight_reader.read_bytes(np.prod(conv_layer.get_weights()[0].shape))\n",
    "            kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))\n",
    "            kernel = kernel.transpose([2,3,1,0])\n",
    "            conv_layer.set_weights([kernel])\n",
    "    return(model)\n",
    "\n",
    "\n",
    "nb_conv = 22\n",
    "model = set_pretrained_weight(model,nb_conv, path_to_weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize the 23rd layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def initialize_weight(layer,sd):\n",
    "    weights = layer.get_weights()\n",
    "    new_kernel = np.random.normal(size=weights[0].shape, scale=sd)\n",
    "    new_bias   = np.random.normal(size=weights[1].shape, scale=sd)\n",
    "    layer.set_weights([new_kernel, new_bias])\n",
    "    \n",
    "layer   = model.layers[-4] # the last convolutional layer\n",
    "initialize_weight(layer,sd=GRID_H*GRID_W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So that's how we define the YOLOv2 model!\n",
    "The next blog will discuss the loss function of this model which will be used to train the parameters.\n",
    "\n",
    "[FairyOnIce/ObjectDetectionYolo](https://github.com/FairyOnIce/ObjectDetectionYolo)\n",
    " contains this ipython notebook and all the functions that I defined in this notebook. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
